Method and apparatus for performing tone scale modifications on a sparsely sampled extended dynamic range digital image

ABSTRACT

A method of generating a tone scale function for a sparsely sampled extended dynamic range digital image, includes the steps of: providing a sparsely sampled extended dynamic range image sensing device having fast photosites with a predetermined response to light exposure interspersed with slow photosites with a slower response to the same light exposure; using the image sensing device to produce a sparsely sampled high resolution digital image having fast pixel values produced by the fast photosites and slow pixel values produced by the slow photosites; and generating the tone scale function using only slow pixel values from the sparsely sampled high resolution digital image.

FIELD OF THE INVENTION

The invention relates generally to the field of image capture, and morespecifically to a method of performing tone scale modifications on anextended dynamic range digital image.

BACKGROUND OF THE INVENTION

Image sensing devices, such as a charge-coupled device (CCD), arecommonly found in such products as digital cameras, scanners, and videocameras. These image sensing devices have a limited dynamic range whencompared to traditional photographic film products. A typical electronicimage sensing device has a dynamic range of about 7 stops. This meansthat the exposure for a typical scene must be determined with a fairamount of accuracy in order to avoid clipping the resultant signal. Bycontrast, natural scenes often exhibit a dynamic range of 9 stops andhigher. This is mainly a consequence of multiple light sources withwidely varying intensities illuminating the scene objects. Specularhighlights also contribute to the dynamic range of natural scenes.

Electronic sensors used to scan photographic film must also contend witha high dynamic range of signal intensities. In U.S. Pat. No. 5,221,848issued Jun. 22, 1993 to Milch entitled High Dynamic Range Film Digitizerand Method of Operating the Same discloses a method and apparatusdesigned to extend the dynamic range of an electronic image sensor.Intended primarily for scanning photographic film, the system describedby Milch includes a one-pass film scanner using a charge-coupled devicescanner having a plurality of linear arrays having the same spectralsensitivity. One of the arrays has a faster response to light than theother array. The information from the two arrays is then combined anddigitized forming an extended dynamic range digital image.

Digital electronic cameras employ a single image sensor with a colorfilter array (CFA) to produce a sparsely sampled digital image. Atypical color filter array pattern is disclosed in U.S. Pat. No.3,971,065 issued Jul. 20, 1976 to Bayer entitled Color Imaging Array.Interpolation algorithms are employed to produce a full resolution colorimage from the sparsely sampled image. Digital cameras also need torecord scenes having a high dynamic range. One way to obtain a highdynamic range image from a digital camera is to employ a high bit depthanalog to digital converter in the camera. Another way is to employ animage sensor having interspersed fast and slow photosites as disclosedin copending U.S. Ser. No. 09/615,398 filed Jul. 13, 2000 by Gallagheret al. which is incorporated herein by reference.

It is known to employ a tone scale function that is specificallydesigned for use with extended dynamic range digital images. Forexample, U.S. Pat. No. 5,822,453 issued Oct. 13, 1998 to Lee et al.entitled Method for Estimating and Adjusting Digital Image Contrastdiscloses a method of calculating a tone scale function using the pixelvalues of an extended dynamic range digital image involving estimatingthe scene contrast from the digital image. However, the method taught byLee et al. is not optimized for images that are captured by extendeddynamic range image sensors having fast and slow photosites.

Therefore, there exists a need for an improved method of performing tonescale modifications on extended dynamic range images of the type thatare captured by extended dynamic range image sensors having fast andslow photosites.

SUMMARY OF THE INVENTION

The need is met according to the present invention by providing a methodof generating a tone scale function for a sparsely sampled extendeddynamic range digital image that includes the steps of: providing asparsely sampled extended dynamic range image sensing device having fastphotosites with a predetermined response to light exposure interspersedwith slow photosites with a slower response to the same light exposure;using the image sensing device to produce a sparsely sampled highresolution digital image having fast pixel values produced by the fastphotosites and slow pixel values produced by the slow photosites; andgenerating the tone scale function using only slow pixel values from thesparsely sampled high resolution digital image.

According to one aspect of the invention, the tone scale function isapplied to the sparsely sampled extended dynamic range digital imageprior to interpolating the digital image to a full resolution digitalimage.

ADVANTAGES

The present invention has the advantage that the tone scale function isgenerated with a subset of pixel values thereby reducing the computationtime for generating the tone scale function.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a digital imaging system utilizing anextended-range image sensing device and a digital image processoraccording to the invention;

FIG. 2 is a functional block diagram of the digital image processor;

FIG. 3 is a cross-sectional view of an interline image sensing deviceemploying an array of lenslets to alter the response of selectedphotosites;

FIG. 4 is a cross-sectional view of a full frame image sensing deviceemploying a metal mask to alter the response of selected photosites;

FIG. 5 is a graph illustrating the response of a fast photosite and aslow photosite;

FIG. 6 is a cross-sectional view of an image sensing device employing anarray of neutral density filters to alter the response of selectedphotosites;

FIG. 7A illustrates the arrangement of slow photosites and fastphotosites on a panchromatic image sensing device;

FIG. 7B illustrates the arrangement of slow photosites and fastphotosites on a color image sensing device;

FIG. 8 Shows a block diagram of the dynamic range extending filter array(DREFA) processor;

FIG. 9 Shows a block diagram of the paxelization module;

FIG. 10 shows an example of the arrangement of pixels for blockaveraging; and

FIG. 11 is a functional block diagram the enhancement processor.

DETAILED DESCRIPTION OF THE INVENTION

A digital image is comprised of a one or more digital image channels.Each digital image channel is comprised of a two-dimensional array ofpixels. Each pixel value relates to the amount of light received by animaging capture device corresponding to the geometrical domain of thepixel. For color imaging applications a digital image will typicallyconsist of red, green, and blue digital image channels. Otherconfigurations are also practiced, e.g. cyan, magenta, and yellowdigital image channels. For monochrome applications, the digital imageconsists of one digital image channel. Motion imaging applications canbe thought of as a time sequence of digital images. Those skilled in theart will recognize that the present invention can be applied to, but isnot limited to, a digital image for any of the above mentionedapplications.

Although the present invention describes a digital image channel as atwo dimensional array of pixel values arranged by rows and columns,those skilled in the art will recognize that the present invention canbe applied to mosaic (non rectilinear) arrays with equal effect. Thoseskilled in the art will also recognize that although the presentinvention describes replacing an original pixel value with processedpixel values, it is also contemplated to form a new digital image withthe processed pixel values and retain the original pixel values.

Imaging devices employing electronic sensing devices are well known,therefore the present description will be directed in particular toelements forming part of, or cooperating more directly with, apparatusin accordance with the present invention. Elements not specificallyshown or described herein may be selected from those known in the art.Note that as used herein, the term image is a two dimensional array ofvalues. An image may be a two dimensional subset of another image. Thepresent invention is preferably implemented using a programmed digitalcomputer. The computer can be a general purpose computer, such as apersonal computer, or a special purpose computer designed for imageprocessing. It is within the ordinary skill of the programming art toproduce a computer program for practicing the present invention based onthe following disclosure.

The essential elements employed in the practice of the present inventionare shown as a functional block diagram in FIG. 1. Light from an objector scene is incident upon a lens 2, forming a photographic image on anextended dynamic range image sensing device 10 such as a charged-coupleddevice (CCD) with a color filter array (CFA) having fast and slowphotosites as described below. Note that other devices, such as CMOSdevices, may be used as the image sensing device 10. The image sensingdevice 10 is a sparsely sampled, extended dynamic range image sensingdevice as will be described in further detail below. An optical low passfilter 6 placed between the lens 2 and the image sensing device 10,performs a slight blurring of the imaged light in order to reduce theoccurrence of aliasing. An A/D converter 14 receives the voltage signalcorresponding to the imaged light from the image sensing device 10 andproduces an image signal corresponding to the voltage signal. Thedigital image processor 200 receives the image signal from the A/Dconverter 14, modifies the image signal and produces a digital image. Asnoted above, the digital image processor 200 can be a programmedpersonal computer, or a special purpose image processor. The user of thedigital imaging system can make selections, with a user selection device62 such as a keyboard, mouse, or button on a camera body, with regard tothe tonal appearance of an image. For example, the user may view animage created with a certain tone scale function on a display device 64.The user may then specify a desired tone scale adjustment. For example,this desired tone scale adjustment may be specified by the userindicating with the user selection device 62 that the shadows should belightened. An adjusted tone scale function is then created using thedesired tone scale adjustment. Creating and modifying a tone scalefunction based on desired tone scale adjustments from a user is wellknown in the art of image processing. For example, commonly assignedU.S. Pat. No. 5,012,333 by Lee et al. describes an interactive methodfor generating a tone scale function and applying the function to animage. The present invention can also be implemented within a digitalcamera. For this digital imaging system application, an enhanced digitalimage derived from a low resolution digital image is displayed on adisplay device 64, for example a liquid crystal display device (LCD) asa component of the digital camera. Alternatively, the present inventioncan be practiced in a film or reflection scanner or other device thatproduces an extended dynamic range digital image.

The A/D converter 14 shown in FIG. 1 converts the voltage signalproduced by the image sensing device 10 into an image signal, i.e. astream of digital pixel values corresponding to the voltage signalproduced by the photosites of the image sensing device 10. Morespecifically, the A/D converter 14 converts the voltage signal, nearlylinear with respect to the intensity of the incident light, from theimage sensing device 10 to a discrete digital image signal, e.g. a 10bit signal where the linear encoded values range from 0 to 1023. The A/Dconverter 14 may also perform processing to convert the linear codevalue domain image signal to a nonlinear code value domain image signal,such as an 8 bit logarithmic signal as is commonly performed in the art.For example, the following equation can be used to convert a 10 bitlinear image signal a(x,y), where (x,y) specifies the row and columnindex of the signal location with reference to the image sensing device10, into an 8 bit logarithmic image signal b(x,y):

$\begin{matrix}{{b\left( {x,y} \right)} = \left\{ \begin{matrix}0 & {0 \leq {a\left( {x,y} \right)} \leq 31} \\{{73.5975\mspace{14mu}\ln\mspace{14mu}{a\left( {x,y} \right)}} - 255} & {32 \leq {a\left( {x,y} \right)} \leq 1024}\end{matrix} \right.} & (1)\end{matrix}$Note that each stop of exposure (in the linear response region of theimage sensing device) results in a doubling of the linear image signala(x,y) and results in an increase of the logarithmically encoded imagesignal b(x,y) by 51. In this case, the value 51 represents the number ofcode values per stop (cvs) of exposure.

The digital image processor 200 shown in FIG. 1 is illustrated in moredetail in FIG. 2. The image signal is received by the dynamic rangeextending filter array (DREFA) processor 22. The DREFA processor 22processes the sparsely sampled high resolution digital image output fromthe A/D converter 14 by expanding the dynamic range of the image andinterpolating the sample values. The DREFA processor 22 then transmitsthe modified image signal to the CFA interpolator 26 where the colorvalues are interpolated to provide a color value at each pixel. Theoutput of the CFA interpolator 26 is a full resolution digital image.The paxelization module 220 also receives the sparsely sampled highresolution digital image and produces a paxelized digital image, i.e. alow resolution digital image derived from the image signal. Optionally,the high resolution digital image is also received by an enhancementprocessor 240 which receives the paxelized digital image, calculatesenhancements based on the paxelized digital image and applies theenhancements to the pixels of the high resolution digital image togenerate an enhanced digital image.

The purpose of the CFA interpolator 26 is to generate a full descriptionof the color for each location of the sensed photographic image. In thepreferred embodiment, the image sensing device 10 consists of an arrayof photosensitive elements called photosites. Each photosite istypically provided with either a red, green, or blue filter, asdescribed by Bayer in commonly assigned U.S. Pat. No. 3,971,065, whichis incorporated herein by reference. The Bayer array is a color filterarray in which green filters are located in a checkerboard pattern overthe photosites with red and blue filters alternating line by line tofill the interstices of the checkerboard pattern; this produces twice asmany green filter sites as either red or blue filter sites. Note thatthe method described herein may be easily extended to color filterarrays with different arrangements of the primaries, a different numberof primaries, or a different set of primaries. Thus, in the preferredembodiment, each photosite is sensitive to either red, green, or bluelight. However, it is desirable to obtain a pixel value corresponding toan exposure for each of the red, green, and blue exposures at eachphotosite location. The pixel values of the sparsely sampled highresolution digital image output from the A/D converter 14 constitute asparsely sampled image having red, green, and blue pixel values atneighboring pixel locations.

In this description, “red”, “green”, and “blue” represent the primaryspectral sensitivities of an image sensing device 10, as is well knownin the art of image processing. The CFA interpolator 26 generates fromthe image signal output from the A/D converter 14 an interpolated imagesignal consisting of a pixel value corresponding to the color primariesfor each photosite. For example, if a particular photosite is coatedwith a red filter, then the A/D converter 14 outputs a red pixel valuefor that photosite since the red filter essentially blocks green andblue light from that particular photosite. The CFA interpolator 26calculates a green pixel value and blue pixel value for thecorresponding photosite even though the corresponding photosite does notrespond to green and blue light. Similarly, the CFA interpolator 26calculates a green pixel value and a red pixel value corresponding tothe blue photosites, as well as a red pixel value and blue pixel valuecorresponding to the green photosites. The operation of the CFAinterpolator 26 can be combined with the DREFA processor 22.Conceptually, the CFA interpolator 26 and the DREFA processor 22 performdistinct operations and for the purpose of clarity are not combined inthis embodiment.

Generally, the CFA interpolator 26 operates by considering the pixelvalues of the corresponding photosite and the pixel values of associatedsurrounding photosites. While any commonly known interpolator may beused, a description of a preferred CFA interpolator is disclosed in.U.S. Pat. No. 5,652,621 issued Jul. 29, 1997 to Adams et al. entitledAdaptive Color Plane Interpolation in Single Sensor Color ElectronicCamera, which is incorporated herein by reference. Adams et al. describean apparatus for processing a digitized image signal obtained from animage sensor having color photosites aligned in rows and columns thatgenerate at least three separate color values but only one color valuefor each photosite location, and a structure for interpolating colorvalues for each photosite location so that it has three different colorvalues. The apparatus generates an appropriate color value missing froma photosite location by the interpolation of an additional color valuefor such photosite locations from color values of different colors thanthe missing color value at nearby photosite locations. The apparatusalso obtains Laplacian second-order values, gradient values and colordifference bias values in at least two image directions from the pixelvalues corresponding to nearby photosites of the same column and row andselects a preferred orientation for the interpolation of the missingcolor value based upon a classifier developed from these values.Finally, the missing color pixel value from nearby multiple color pixelvalues is selected to agree with the preferred orientation.

The sparsely sampled high resolution digital image output from the A/Dconverter 14 is received by the DREFA processor 22 which expands thedynamic range of the image signal to create a modified image signalhaving an expanded dynamic range. In the preferred embodiment, thedynamic range of the image sensing device 10 is expanded by designingcertain photosites of the image sensing device 10 to have a slowresponse. The arrangement of the slow response photosites with respectto the image sensing device 10 will be discussed in greater detailhereinbelow. The responses of the slow response photosites are slowed,or retarded, by altering the gain of the selected photosites, hereinreferred to as slow photosites. Altering the gain of a photosite iscommonly practiced in the art of digital camera design and manufacture.

With reference to FIG. 3, it is a common practice in the art of imagesensor manufacture to place resin lenslets 51 on top of each photosite.For example, particularly when the image sensing device 10 is aninterline solid state image sensing device, one such lenslet techniqueis described in U.S. Pat. No. 4,667,092 issued May 19, 1987 to Ishiharaentitled Solid-State Image Device with Resin Lens and Resin ContactLayer, which is incorporated herein by reference. Ishihara discloses asolid-state image device which includes an image storage block having ablock surface and a plurality of storage elements embedded along theblock surface to store an image in the form of electric charge. Anoverlying layer is deposited to form an array of optical lenses incorrespondence to the storage elements. An intermediate layer is laidbetween the block surface and the overlying layer. Incident lightfocuses through the lenses and the intermediate layer onto the storageelements. The intermediate layer serves as an adjusting layer foradjusting the focal length of the lenslets.

FIG. 3 shows a cross section of an interline solid state image sensingdevice. Without the lenslets 51, the signal readout area associated witheach photosensitive area 55 of a photosite makes it impossible to usethe whole area of the semiconductor substrate as the photoelectrictransducer area. The conventional solid-state image device does noteffectively utilize all incident rays thereon and therefore has lowsensitivity. The addition of a resin lenslet 51 on top of a photositeallows the incident rays of light to be focused on the photoactive areasof the photosite, thereby more effectively utilizing the incident raysof light and increasing the sensitivity of the photosite. Thus, byvarying the size and/or efficiency of the lenslet 51, the sensitivity(gain) of the photosite may be easily altered. Thus, for interlinedevices and for CMOS sensors the preferred method of altering the gainof the photosite is by altering the lenslet 51 placed on top of thephotosite. As shown in FIG. 3, the location 52 has no lenslet, andtherefore fewer incident rays of light are incident on thephotosensitive area. Alternatively, a lenslet could be manufactured atlocation 52 with a different radius, shape, size or material as comparedwith the lenslet 51, thereby structured to be less efficient at focusingincident rays of light onto the photosensitive area 55 than is thelenslet 51. Those skilled in the art will recognize that if the lenslet51 focuses 80% of the incident rays of light onto a photosensitive area55 and the location 52 having no lenslets (or alternatively slowlenslets) allows 20% of the incident rays of light onto a photosensitivearea 55, then the photosite covered by lenslet 51 is 2 stops faster thanthe location 52. In this case, the lenslet 51 is used for fastphotosites and no lenslet is used for slow photosites, as represented bylocation 52.

With reference to FIG. 4 showing a cross section of a full frame imagesensing device 10, in the case where the image sensing device 10 is afull frame device, light rays incident to the photosensitive area 55 ofa photosite must pass through an aperture of a light shield, typicallymade from metal, which is shown in cross-section in FIG. 4 to compriselight blocking mask portions 54 and large and small apertures 56 and 57interspersed among the light blocking portions. In the preferredembodiment, the gain of photosites may be altered by modifying the lightblocking mask portion 54. The sensitivity of the photosite is thendirectly related to the aperture of the light blocking mask portion 54.For example, one photosite with an aperture 50% of the size of a secondphotosite's aperture will have a response of 50% compared to that on thesecond photosite. For example, a large aperture 56 of a light blockingmask portion 54 allows 80% of the light rays incident upon thatphotosite to pass through, but a small aperture 57 allows only 20% ofthe incident light rays to pass. Those skilled in the art will recognizethat the photosite with the large aperture 56 is 2 stops faster than aphotosite with the small aperture 57. In this case, the large aperture56 is used for fast photosites, and the small aperture 57 is used forthe slow photosites. Thus, the aperture of the light blocking mask maybe modified to adjust the response of the selected photosites. TheEastman Kodak Company makes full frame image sensing devices with ametal mask light shield that reduces the pixel active area of all pixelsfrom about 80% to about 20% (for dithered scanner applications where thesensor is moved by ½ the pixel spacing horizontally and vertically, and4 pictures are taken). The technique thus involves utilizing such masktechnology, but with different sized apertures, to provide an imagesensor with a differential response to image light.

In the preferred embodiment, the response of the selected slowphotosites is X% (where X<=100) that of fast photosites for the sameexposure, as shown graphically in FIG. 5. In this preferred embodiment,the selected photosites have a response that is slowed by two stops(−log X/100) relative to the fast photosites, resulting in X=25. Thus,the image sensing device 10 consists of multiple sets of photosites,fast photosites and slow photosites. The collection of the outputresponses of the fast photosites constitutes a sparsely sampled fastdigital image, i.e. a sparsely sampled version of a scene sensed withthe fast photosites. Likewise, the collection of the output responses ofthe slow photosites constitutes a sparsely sampled slow digital image,i.e. a sparsely sampled version of a scene sensed with the slowphotosites.

As another alternative, the responses of the selected slow photositescan be slowed by the use of a neutral filter coating the photosite. FIG.6 shows a cross section of an image sensing device with a color filterarray 53. Note that the color filter array 53 a is red, 53 b is green,53 c is red, and 53 d is green. A layer of neutral filters 58 iscontained above the color filter array 53, although the position of thelayer of neutral filters 58 and the color filter array 53 does notmatter. Note that the layer of neutral filters 58 only contains aneutral filter at the positions of selected photosites, as indicated bythe neutral filter 59. In this case, the layer of neutral filters 58 istransparent or nearly transparent for fast photosites and contains aneutral filter 59 for slow photosites. For example, if the neutralfilter 59 consists of a material that allows X% transmission of light,then the response of that slow photosite will be slowed by

$- {\log_{2}\left( \frac{X}{100} \right)}$stops relative to the response of the fast photosite.

The DREFA processor 22 shown in FIG. 2 is described in more detailhereinbelow. The purpose of the DREFA processor 22 is to create amodified image signal with an increased dynamic range by processing theinput image signal while accounting for the difference in photo responseof the fast and slow photosites. Accordingly, the output of the DREFAprocessor 22 is a modified image signal having increased numericaldynamic range. This modified image signal is then input to the CFAinterpolator 26 for processing as previously described.

It is not a requirement of the present invention that the A/D converter14 and the DREFA processor 22 be directly connected. The DREFA processor22 may reside in hardware or software in close proximity to the A/Dconverter 14 and image sensing device 10. For example, the DREFAprocessor 22 could reside directly within a digital camera. However, theDREFA processor 22 may also be remote from the image sensing device 10.For example, the image signal output from the A/D converter 14 can betransmitted (after compression) via a wire or wireless connection to apersonal computing device, printer, or remote server to apply tooperation of the DREFA processor 22. Transmission of the image signalmay also include file transfer protocol or email.

In the preferred embodiment, 50% of the photosites of the image sensingdevice 10 are selected to have slow response. Those skilled in the artwill recognize that the relative percentages of slow and fast pixels isnot critical, and that the advantages of the present invention can beachieved with various relative percentages of fast and slow photosites.In the case of an image sensing device 10 in which all photosites haveapproximately equivalent spectral sensitivity (i.e. a panchromatic imagesensing device), FIG. 7A shows an arrangement of the slow photositesthat will result in approximately 50% of all the photosites of the imagesensing device 10 being of slow response. The photosites 28 with slowresponse are marked with an asterisk (*), while the photosites 30 havingfast response are blank. A sparsely sampled image was previously definedas an image that was captured with an image sensing device 10 having acolor filter array. According to the present invention, the termsparsely sampled is also intended to refer to an image produced by animage sensing device such as that shown in FIG. 7A where the fast andslow photosites are interspersed. Additionally, an image sensing device10 such as shown in FIG. 7A having fast photosites with a predeterminedresponse to light exposure interspersed with slow photosites having aslower response to the same light exposure is a sparsely sampledextended dynamic range image sensing device.

FIG. 7B shows an arrangement for a color image sensing device 10 wherein50% of each photosite type (red, green, or blue sensitive) has slowresponse. For example, the photosites 32, 34, and 36 are red, green andblue photosites, respectively, having slow responses; the photosites 38,40, and 42 are red, green and blue photosites, respectively, having fastresponses. Note that the image sensing device 10 is also a sparselysampled extended dynamic range image sensing device according to theprevious definition.

Note that FIGS. 7A and 7B imply a regular pattern for the location ofthe slow photosites. While it is preferable that the slow photosites arearranged in a regular pattern, it is by no means necessary. The slowphotosites could be arranged randomly or semi-randomly over the surfaceof the image sensing device 10, and their location would be stored insome place accessible to the DREFA processor 22.

Referring to FIG. 5, the response of a fast photosite to a certainexposure and the response of a slow photosite to the same exposure areshown. Note that if a level of noise n is superimposed on the response,it can easily be seen that the fast photosite will yield a valid signalwith lower exposures (beginning at exposure level E) than will the slowphotosite (which yields valid signal beginning at

$\left. {\frac{100}{X}{E.}} \right)$Alternatively, data from the slow photosite will be valid for higherexposure levels (up to signal level of

${\frac{100}{X}{E2}^{S}},$where S is the inherent dynamic range of a single photosite, typically Smay be about 5 stops) than would the fast photosite (which producesvalid response up to an exposure of E2 ^(S)). Note that both the fastphotosite and the slow photosite have the same range of response instops of exposure (S), but the response of the slow photosites ispreferably

$- {\log_{2}\left( \frac{X}{100} \right)}$stops slower than the fast photosites, as shown in FIG. 5. It ispreferred that the responses of the fast and slow photosites overlapwith respect to exposure. That is, it is preferred that

${- {\log_{2}\left( \frac{X}{100} \right)}} < {S.}$The overall dynamic range of the image sensing device 10, consideringboth fast and slow photosites, is

$S - {{\log_{2}\left( \frac{X}{100} \right)}.}$In the case of the preferred embodiment, where S=5 and X=25, the overalleffective dynamic range of the image sensing device 10 is 7 stops ofexposure.

The DREFA processor 22 may be utilized to extend the overall dynamicrange of the digital images produced with the present invention by usingthe pixel values corresponding to slow photosites to reconstruct theimage signals in regions corresponding to very high exposures. Likewise,the DREFA processor 22 also uses the pixel values corresponding tophotosites with fast response to reconstruct the image signalcorresponding to very low exposures.

FIG. 8 shows a block diagram of the DREFA processor 22. The sparselysampled high resolution digital image, which is a logarithmic imagesignal b(x,y) output from the A/D converter 14, is passed to the slowpixel compensator 44. The purpose of the slow pixel compensator 44 is tocompensate the image signal corresponding to slow photosites byaccounting for the offset in response by X stops. Alternatively, thefast pixels can be equalized to the slow pixels by adjusting the fastpixels in the opposite direction. In the preferred embodiment, the imagesignal corresponding to the slow photosites are incremented by thequantity −cvs log(X/100), where cvs is the number of code values perstop of exposure. In the preferred embodiment, the quantity cvs is 51.Alternatively, if the image signal input to the slow pixel compensator44 is linearly related to exposure (rather than logarithmically), thenthe slow pixel compensator 44 scales the image signal corresponding tothe slow photosites by a factor of 100/X . Note that it is assumed thatthe locations of the slow photosites are known to the slow pixelcompensator 44. The output of the slow pixel compensator 44 is an imagesignal i(x,y) that has been compensated at the locations ofcorresponding to slow photosites for the difference between the slowphotosite response in relation to the fast photosite response. At thelocations corresponding to fast photosites, the value of the imagesignal b(x,y) output from the A/D converter 14 is identical to the valueof the image signal i(x,y) output from the slow pixel compensator 44.Note that the image signal i(x,y) is not limited to an 8 bit range. Inthe preferred embodiment, the value of i(x,y) ranges from 0 to 357 (i.e.9 bits).

Next, the image signal i(x,y) output from the slow pixel compensator 44is input to a slow pixel thresholder 46. The purpose of the slow pixelthresholder 46 is to determine slow pixel values that are of low qualitydue to a photosite not receiving enough photons to produce a validsignal. The pixel value at these (x,y) locations is then replaced inprocessing performed by the signal extender 50 by calculating a newpixel value based upon nearby fast pixel values. All slow pixel valueswhich are less than a predetermined threshold are considered to beproblem pixel values. In the case of the slow pixel values, thispredetermined threshold is referred to as the low exposure responsethreshold. Thus, a pixel value i(x,y) is considered to be a problem ifit is a slow photosite and if:i(x,y)<T₁  (2)where T₁ is predetermined. In the preferred embodiment, the value of T₁is given by

$\begin{matrix}{{T_{1} = {{- {cvs}}\mspace{14mu}{\log_{2}\left( \frac{X}{100} \right)}}},} & (3)\end{matrix}$which in the preferred embodiment is set to a value of 102. Note thatthe threshold T₁ may be dependent upon the color sensitivity of thephotosite at location (x,y). Slow pixel values that are problems arereferred to as noise pixels, since the value of i(x,y) is notsufficiently above the noise level of the image sensing device to beuseful.

Likewise, the image signal i(x,y) output from the slow pixel compensator44 is input to a fast pixel thresholder 48. The purpose of the fastpixel thresholder 48 is to determine fast pixels that are of lowquality. The pixel values at these locations is then replaced bycalculating a new pixel value based upon nearby slow pixel values inprocessing performed by the signal extender 50, which will be describedin detail hereinbelow. All fast pixel values that are greater than apredetermined threshold value are considered to be problem pixels. Inthe case of the fast pixels, this predetermined threshold used for thepurpose of detecting problem fast pixels is referred to as the highexposure response threshold. Thus, a fast pixel value i(x,y) isconsidered to be a problem if:i(x,y)>T₂  (4)where T₂ is a predetermined threshold. In the preferred embodiment, thevalue of T₂ is set to a value of 254. Note that the threshold T₂ may bedependent upon the color of the photosite at location (x,y). Fastphotosites that are problem locations are referred to as saturatedpixels, since the value of i(x,y) is as high as possible at theselocations.

The (x,y) locations of the problem slow pixels determined by the slowpixel thresholder 46 and the (x,y) locations of the problem fast pixelsdetermined by the fast pixel thresholder 48 are input to the signalextender 50. In addition, the image signal i(x,y) output from the slowpixel compensator 44 is also input to the signal extender 50. Thepurpose of the signal extender 50 is to replace the image signal i(x,y)values at problem locations (xy) with estimates of the signal hereinreferred to as replacement values, had the inherent dynamic range ofeach photosite of the image sensing device 10 been greater. If theproblem location is coincident with a slow photosite, then thereplacement value is calculated from neighboring image signal pixelvalues coincident with fast photosites. In this embodiment, the term“neighboring” refers to a certain spatial distance. In the preferredembodiment, the photosites neighboring a selected photosite are thosephotosites within a distance of 2 photosites of the selected photosite.Likewise, if the problem location is coincident with a fast photosite,then the replacement value is calculated from neighboring image signalvalues coincident with slow photosites. In the preferred embodiment, thecolor of the photosite at the problem photosite is also taken intoaccount. The replacement value for any problem location is preferablydetermined only by the signal originating from neighboring photosites ofthe same color. The output of the signal extender 50 is an image signali′(x,y) having a dynamic range as if captured by an image sensing device10 having photosites with inherent dynamic range of

$\begin{matrix}{S = {- {\log_{2}\left( \frac{X}{100} \right)}}} & (5)\end{matrix}$rather than the actual inherent dynamic range of S for each photosite ofthe image sensing device 10. Note that for all (x,y) locations that arenot problem locations, the value of i′(x,y) is equivalent to i(x,y).

As an example of the processing performed by the signal extender 50 forthe Bayer CFA pattern shown in FIG. 7B, if location (x,y) is a problemlocation, and (x,y) is the location that corresponds to a greenphotosite (such as photosite 34 in FIG. 7B), then the replacement valuei′(x,y) for the image signal i(x,y) is calculated in the followingmanner:i′(x,y)=0.25*[i(x−1,y−1)+i(x+1,y−1)+i(x−1,y+1)+i(x+1,y+1)]  (6)Note that signal values that the calculation of i′(x,y) is dependentupon, are expected to comply with certain requirements. For example,suppose that (x,y) is a problem location and (x,y) corresponds to agreen photosite with slow response. Then the signal levels ofneighboring photosites are used to calculate replacement value i′(x,y).However, this assumes that the signal values of each of the neighboringphotosites are also less than T₃. In the preferred embodiment, T₃=T₁.For each neighboring photosite that this is not the case, that signallevel is omitted from the calculation of the replacement value i′(x,y).For example, if i(x−1, y−1)>T₃, then the value i′(x,y) is calculatedwith the following formula:i′(x,y)=⅓*[i(x+1,y−1)+i(x−1,y+1)+i(x+1,y+1)]  (7)Generally, the interpolation scheme for determining a replacement valueat problem location (x,y), where the location (x,y) corresponds to agreen photosite which is also a fast photosite on a image sensing devicehaving a Bayer pattern filter array is given with the followingequation:

$\begin{matrix}{{i^{\prime}\left( {x,y} \right)} = \frac{\sum\limits_{{j = {- 1}},1}\;{\sum\limits_{{k = {- 1}},1}\;{{i\left( {{x + j},{y + k}} \right)}{W\left( {{x + j},{y + k}} \right)}}}}{\sum\limits_{{j = {- 1}},1}\;{\sum\limits_{{k = {- 1}},1}\;{W\left( {{x + j},{y + k}} \right)}}}} & (8)\end{matrix}$where

$\begin{matrix}{{W\left( {{x + j},{y + k}} \right)} = \left\{ \begin{matrix}1 & {{i\left( {{x + j},{y + k}} \right)} > T_{3}} \\0 & {otherwise}\end{matrix} \right.} & (9)\end{matrix}$Note that the same equation is applied to determine the replacementvalue if the problem location corresponds to a green photosite which isalso a slow photosite. However, in this case:

$\begin{matrix}{{W\left( {{x + j},{y + k}} \right)} = \left\{ \begin{matrix}1 & {{i\left( {{x + j},{y + k}} \right)} > T_{4}} \\0 & {{otherwise},}\end{matrix} \right.} & (10)\end{matrix}$where in the preferred embodiment, T₄=T₂.

As another example, also in connection with the Bayer CFA pattern shownin FIG. 7B, if location i(x,y) is a problem photosite and (x,y)corresponds to a location of a red or blue photosite, then thereplacement value i′(x,y) for the image signal i(x,y) is calculated inthe following manner:i(x,y)=0.25*[i(x−2,y)+i(x+2,y)+i(x,y+2)+i(x,y−2)].  (11)When location (x,y) corresponds to a red or blue photosite and is also afast photosite, the equation for determining the replacement valuei′(x,y) may be generalized as follows:

$\begin{matrix}{{i^{\prime}\left( {x,y} \right)} = \frac{\sum\limits_{{j = {- 2}},0}\;{\sum\limits_{{k = {- 2}},0,2}\;{{i\left( {{x + j},{y + k}} \right)}{W\left( {{x + j},{y + k}} \right)}}}}{\sum\limits_{{j = {- 2}},0,2}\;{\sum\limits_{{k = {- 2}},0,2}\;{W\left( {{x + j},{y + k}} \right)}}}} & (12)\end{matrix}$where

$\begin{matrix}{{W\left( {{x + j},{y + k}} \right)} = \left\{ \begin{matrix}1 & {{i\left( {{x + j},{y + k}} \right)} > T_{3}} \\0 & {otherwise}\end{matrix} \right.} & (13)\end{matrix}$Note that in this case, either j or k must be 0, but j and k are neverboth zero. Note also that the same equation is applied to determine thereplacement value if the problem location corresponds to a red or bluephotosite which is also a slow photosite. However, in this case

$\begin{matrix}{{W\left( {{x + j},{y + k}} \right)} = \left\{ \begin{matrix}1 & {{i\left( {{x + j},{y + k}} \right)} > T_{4}} \\0 & {{otherwise},}\end{matrix} \right.} & (14)\end{matrix}$where in the preferred embodiment, T₄=T₂.

The interpolation scheme described above for the purpose of generatingan image signal with an extended dynamic range from more than onesparsely sampled image signal may be modified by those skilled in theart. However, many such modifications to the above interpolation schememay be derived and should not be considered as significant deviations ofthe present invention.

Those skilled in the art will recognize that the above interpolationscheme performed by the signal extender is a lowpass filter, which iswell known in the art. Typically, the application of a lowpass filter toan image signal has a similar effect to reducing the resolution of theimage signal. Thus, the processing performed by the DREFA processor 22is a method by which the spatial resolution of the image sensing device10 may be traded for dynamic range of the image sensing device 10.Indeed, those areas of an image where the interpolation scheme isimplemented to increase the dynamic range of the signal appearnoticeably softer (less sharp) than the image would have if that samearea of the image had been captured by the image sensing device in sucha fashion that no “problem locations” occur (as defined by the slowpixel thresholder 46 and the fast pixel thresholder 48).

The paxelization module 220 shown in FIG. 2 is illustrated in moredetail in FIG. 9. The purpose of the paxelization module 220 is tocreate a paxelized digital image for the purpose of analysis byalgorithms in order to determine density and color balance and determinea desired tone scale function by the enhancement processor 240 as willbe described in detail hereinbelow. The paxelization module 220 receivesan image signal from the A/D converter 14. The image signal is receivedby the pixel type separator 222. The purpose of the pixel type separator222 is to allow access to each type of pixel separately. In the presentinvention, the photosites are of two types, fast or slow as shown inFIG. 7A in the case of a monochromatic image sensing device 10.Furthermore in the case of the color image sensing device 10 as shown inFIG. 7B, within each category, the photosite may also be red, green, orblue. (Those skilled in the art will recognize that many other color andspeed combinations are possible.) Thus there are a total of six types ofphotosites. In this embodiment, the pixel type separator 222 separatelyoutputs all the fast pixel values and all the slow pixel values. Thepaxelization engine 224 receives the slow pixel values and performs anumerical averaging spatial filtering technique which results in apaxelized digital image. In the preferred embodiment, the paxelizationengine 224 utilizes block averaging with block size N×N in order tocreate the paxelized digital image which is output from the paxelizationmodule 220. For example, if a block size of N=32 pixels is used by thepaxelization engine 224, then when the sparsely sampled high resolutiondigital image has a resolution of 1024×1536, then the paxelized digitalimage will have a resolution of 32×48 as illustrated in FIG. 10.Typically the value of N is selected such that the paxelized digitalimage is of small size, such as is used by color and tone analysisalgorithms. Each pixel of the paxelized digital image corresponds to aN×N block of the sparsely sampled high resolution digital image. Eachpixel value of the paxelized digital image has associated pixel valuefor each color of the digital image, typically red, green, and blue. Thered pixel values of the paxelized digital image are determined by thepaxelization engine 224 by averaging all of the red pixel values withinthe corresponding N×N block of the sparsely sampled high resolutiondigital image. Notice that since in the preferred embodiment the pixeltype separator 222 outputs only slow pixel values, each pixel of thepaxelized digital image will be calculated exclusively with slow pixelvalues. A similar process is followed for the green and the blue pixelvalues.

It is important to note that the action taken by pixel type separator222 does not have to rearrange the storage of the pixel data in computermemory. The present invention implements the pixel type separator 222 asa pointer addressing scheme to the storage of the pixel data in computermemory. Thus the most important aspect of the pixel type separator 222is capability of indexing pixel data corresponding to the fastphotosites and slow photosites as a separate entities.

An alternative embodiment of the pixel type separator 222 generates apaxelized image having only a single pixel value at each pixel location.This pixel value can be thought of as a luminance pixel value. Theluminance pixel values of the paxelized digital image can be obtained byusing only the slow green pixel values within the corresponding N×Nblock of the sparsely sampled high resolution digital image.Alternatively, luminance pixel values may be derived by using all of theslow pixel values within the corresponding N×N block of the sparselysampled high resolution digital image. As a further alternativeembodiment illustrated in FIG. 2, the paxelization module 220 may usesimilar techniques to generate a paxelized digital image from the imagesignal output from the DREFA processor 22 or from the full resolutiondigital image output from the CFA interpolator 26.

The low resolution module 232 shown in FIG. 2 creates a low resolutiondigital image for the purpose of allowing a user to interactivelydetermine balance and tone scale function modifications by allowing theuser to specify desired tone scale adjustments. The user inputs thesedesired tone scale adjustments with the user selection device 62, shownin FIG. 1. The operation of the low resolution module 232 (see FIG. 2)is similar to the paxelization module 220 except that the factor of N isgenerally selected so the low resolution image is appropriate for thedisplay device 64. Generally, the factor of N for the low resolutionmodule 232 is greater than the factor of N for the paxelization module220. In the preferred embodiment, the factor of N used to generate thelow resolution digital image is 8. If the display device 64 is a colordisplay device, then the low resolution digital image should have morethan one color pixel value for each pixel location. Alternatively, ifthe display device 64 is not a color display device, then the lowresolution digital image may have only a single luminance pixel value aspreviously described.

The enhancement module 240 shown in FIG. 2 is illustrated in more detailin FIG. 11. The full resolution digital image output from the CFAinterpolator 26 is input to the enhancement processor. In an alternativeembodiment, the position of the enhancement processor 24 may be placedin from of the CFA interpolator 26 or the DREFA processor 22.

The luminance-chrominance module 320 ₁ receives the full resolutiondigital image and generates a full resolution LCC digital image. Theluminance-chrominance module 320 ₂ receives the low resolution digitalimage and generates a low resolution LCC digital image. In applicationssuch as monochrome imaging, the luminance chrominance module 320 may beomitted.

The scene balance module 310 receives the paxelized digital image andcalculates balance parameters which relate to an overall in color andbrightness change to be imparted to final processed digital image. Thepaxelized digital image and the balance parameters are received by thetone scale function generator 330 which calculates a lightness tonescale function, i.e. a single valued function used for transformingpixel values. The tone scale function applicator 340 ₂ receives thelightness tone scale function and the low resolution LCC digital imageand applies the lightness tone scale function to the low resolution LCCdigital image to produce a tone scale modified low resolution digitalimage. The tone scale modified low resolution digital image representsan extended dynamic range digital image which has been balanced forbrightness and color and had its tone scale adjusted to improve shadowand highlight detail. The tone scale modified low resolution digitalimage is received by the rendering module 350 which produces a lowresolution rendered digital image. The process of converting the colorvalues captured by an image sensing device to those appropriate fordisplay on a particular output device is often referred to as“rendering”. The low resolution rendered digital image is then displayedon a display device 64 for the purpose of gathering user feedback. Theuser inputs desired changes to the tone scale function using a userselection device 62 such as a button, mouse, touch screen, slider, voicecommands, or other methods of inputting the desired tone scaleadjustments. As an example of the desired tone scale adjustments, theuser indicates by way of the user selection device 62 that thehighlights of the image on the display device 64 should be darkened. Thescene balance module 310 and the tone scale function generator 330 thengenerate modified balances and tone scale functions, respectively.Making image balance and tone scale dependent on user interaction iswell known in the art. For example, U.S. Pat. No. 5,012,333 issued Apr.30, 1991 to Lee et al. entitled Interactive Dynamic Range AdjustmentSystem for Printing Digital Images describes an interactive method forsetting then applying an image dependent tone scale function. The tonescale function applicator 340 ₂ receives the modified tone scalefunction and the low resolution LCC digital image and applies themodified tone scale function to the low resolution LCC digital image toproduce a tone scale modified low resolution digital image. Note thatseveral methods exist for applying a tone scale function to a digitalimage and will be described in greater detail hereinbelow. Afterrendering by the rendering module 350, the low resolution rendereddigital image is displayed to the display device 64 in order to allowthe user to make further adjustments with the user selection device 62.This process may repeat itself until the user has no furtheradjustments. In that case, the tone scale function applicator 340 ₁applies the tone scale function (or the modified tone scale function inthe case where the user makes no adjustments) to the full resolution LCCdigital image in order to produce a tone sale adjusted full resolutionLCC digital image. The RGB conversion module 360 converts this LCCdigital image back to an image represented with red, green, and bluepixel values to produce a tone scale adjusted full resolution digitalimage.

In an alternative embodiment, in some applications such as high speedapplications, the display device 64 and user selection device 62 may beomitted. In this case, the low resolution digital image need not becreated, and the tone scale function output from the tone scale functiongenerator 330 is applied to the full resolution LCC digital image by thetone scale function applicator 340 ₁.

The luminance-chrominance module 320 ₁ is used to generate a fullresolution LCC digital image from the full resolution digital image. Thefull resolution digital image includes red, green, and blue digitalimage channels. Each digital image channel contains the same number ofpixels. Let R_(ij), G_(ij), and B_(ij) refer to the pixel valuescorresponding to the red, green, and blue digital image channels locatedat the i^(th) row and j^(th) column. Let L_(ij), C1_(ij), and C2_(ij)refer to the transformed pixel values of the modified digital image. The3 by 3 matrix transformation relating the input and output pixel valuesis as follows:L_(ij)=τ₁₁R_(ij)+τ₁₂G_(ij)+τ₁₃B_(ij)C1_(ij)=τ₂₁R_(ij)+τ₂₂G_(ij)+τ₂₃B_(ij)C2_(ij)=τ₃₁R_(ij)+τ₃₂G_(ij)+τ₃₃B_(ij)  (15)where the τ_(mn) terms are the coefficients of the 3 by 3 LCC matrixdenoted by [τ]. The constants employed by the present invention for τ₁₁,τ₁₂ and τ₁₃ are 0.333, 0.333 and 0.333 respectively. It is important tonote that the present invention may be practiced with other luminancechrominance transformations and still yield good results. For example, amatrix with τ₁₁, τ₁₂ and τ₁₃ values set to 0.30, 0.59, and 0.11respectively also works well. The calculated values C1 and C2 areexamples of digital color difference values. The luminance-chrominancemodule 320 ₂ is also used to generate, in identical manner, a lowresolution LCC digital image.

The paxelized digital image output from the paxelization module 220 isinput to the scene balance module 310. The present invention may bepracticed with any scene balance module 310 such as the one described inU.S. Pat. No. 4,945,406 issued Jul. 31, 1990 to Cok entitled Apparatusand Accompanying Methods for Achieving Automatic Color Balancing in aFilm to Video Transfer System. The scene balance module calculatesbalance parameters, i.e. pixel values corresponding to the pixel valuesof theoretical 18% gray card, based on the pixel values of the paxelizeddigital image. In the method taught by Cok, statistical parameters arecalculated from the pixels of the paxelized digital image such as pixelminimum, maximum and average values. These statistical parameters arecalculated for different regions of the paxelized digital image and forthree color digital image channels (L, C1 and C2). The statisticalparameters are combined in a multilinear equation with predeterminedcoefficients to predict three balance parameters, i.e. an L channelbalance parameter relating to the overall image brightness and twochrominance balance parameters for the C1 and C2 digital image channelswhich relate to the overall color balance of the image. Note that thesecolor balance values, although expressed in terms of L, C1 and C2 may beeasily converted to other color spaces (e.g. red, green, and blue) bytechniques commonly known to those in the field of color science.

The tone scale function generator 330 receives the paxelized digitalimage and calculates a tone scale function, i.e. a single valuedmathematical equation or transformation that has a single output valuecorresponding to each input value. The present invention implements thelightness tone scale function as a look-up-table for computationefficiency. The present invention may be practiced with a variety ofmethods which generate tone scale functions. The preferred embodiment ofthe present invention uses the methods disclosed in U.S. Pat. No.4,731,671 issued Mar. 15, 1988 to Alkofer entitled Contrast Adjustmentin Digital Image Processing Method Employing Histogram Normalization andU.S. Pat. No. 5,822,453, referenced above. These methods are employed bythe present invention to produce two individual tone scale functions.These two tone scale functions are then cascaded into single lightnesstone scale function which is used to adjust the brightness and contrastof the LCC digital image.

In U.S. Pat. No. 5,822,453, Lee et al. disclose a method of calculatinga tone scale function using the pixel values of a digital imageinvolving estimating the scene contrast from the digital image. Themethod taught by Lee et al. involves calculating a Laplacian filteredversion of the digital image, forming a histogram of the Laplaciansignal, determining from the Laplacian histogram, two threshold valueswhich when applied to the Laplacian signal substantially eliminateuniform areas; sampling pixels from the digital image which are based onthe thresholds; forming a histogram from the sampled pixels; computing astandard deviation of the sampled histogram; and estimating contrast ofthe digital image by comparing the computed standard deviation with apredetermined contrast for determining contrast of the input image inrelationship with the predetermined contrast. The method described byLee and Kwon is used to calculate a first tone scale function.

In U.S. Pat. No. 4,731,671, Alkofer discloses a method of calculating atone scale function using the pixel values of digital image based onnormalizing the histogram of a digital image. This method involvesdetermining the contrast of the digital image by calculating thestandard deviation of a sample of pixel values. The second tone scalefunction is calculated by normalizing a histogram of the sample ofpixels values. The sample of pixel values is selected from one of aplurality of samples of pixel values corresponding to a plurality ofcontrast intervals based upon the shape of the histogram of the selectedsample of pixel values. To facilitate the adjustment of contrast, thetone scale function is constructed to produce values in units of astandard normal variate Z. These Z values are then multiplied by aconstant which is a function of the standard deviation of the sample ofpixel values to determine the contrast of the processed digital image.The present invention uses the L (luminance) digital image channel ofthe LCC paxelized digital image as a base for analysis of the secondtone scale function.

The first and second tone scale functions are combined into a lightnesstone scale function using the mathematical formula:LUT_(f)[j]=LUT₁ [LUT₂[j]]  (16)where LUT₂ represents the second tone scale function, LUT₁ representsthe first tone scale function, and LUT_(f) represents the lightness tonescale function. The j variable represent the index of pixel values ofthe digital image to be processed. The lightness tone scale functionLUT_(f) is calculated by evaluating the expression of equation (16) forthe range of possible pixel values.

The lightness tone scale function LUT_(f) and the LCC digital image arereceived by the tone scale function applicator 340 ₁, and 340 ₂. Thepresent invention applies the lightness tone scale function to theluminance channel of the full resolution LCC digital image and the lowresolution LCC digital image respectively to adjust the brightness andcontrast attributes of the source digital image. Applying the lightnesstone scale function directly to the L (luminance) channel of the LCCdigital image is the fasted method. However, the present invention maybe practiced with other methods of applying the lightness tone scalefunction to the pixels of a digital image. For example, an alternativeembodiment is described in U.S. Ser. No. 09/163,645 filed Sep. 30, 1998by Gallagher et al. Alternatively, commonly assigned U.S. Pat. No.5,012,333 also describes a method of applying a tone scale function to adigital image. These methods involve spatially filtering the luminancedigital image channel resulting in two spatial frequency components(high and low components), applying the tone scale function to the lowspatial frequency component, and combining the tone scale modified lowspatial frequency component with the high spatial frequency component.The resulting processed digital image has enhanced brightness andcontrast attributes with improved spatial detail content. Both of thesemethods utilize spatial filters in order to apply the tone scalefunction.

The RGB conversion module 360 receives tone sale adjusted fullresolution LCC digital image and transforms the luminance-chrominancerepresentation back to a red-green-blue channel representation with theapplication of a linear 3 by 3 matrix transform. The resulting digitalimage channels of the tone scale adjusted full resolution digital imagehave the same color metric representation as the source digital image.The transformation generates new pixel values as linear combinations ofthe input color pixel values.

Let L_(ij), C1_(ij), and C2_(ij) refer to the pixel values correspondingto the luminance and two chrominance digital image channels located atthe i^(th) row and j^(th) column. Let R′_(ij), G′_(ij), and B′_(ij)refer to the transformed pixel values of the modified digital image. The3 by 3 matrix transformation relating the input and output pixel valuesis as follows:R′_(ij)=η₁₁L_(ij)+η₁₂C1_(ij)+η₁₃C2_(ij)G′_(ij)=η₂₁L_(ij)+η₂₂C1_(ij)+η₂₃C2_(ij)B′_(ij)=η₃₁L_(ij)+η₃₂C1_(ij)+η₃₃C2_(ij)  (17)where the η_(mn) terms are the coefficients of the 3 by 3 matrixtransformation.

The preferred embodiment of the present invention constructs the rgbconversion matrix denoted by [η] above as the inverse of the lcc matrixdenoted by [τ] corresponding to the luminance-chrominance module 320.This is mathematically represented in matrix notation in equation (18).[η]=[τ]⁻¹  (18)

The rendering module 350 shown in FIG. 11 receives tone scale modifiedlow resolution digital image, applies a transform to the pixel values,and produces a low resolution rendered digital image that is directlyviewable on a display device. Two transforms are used to prepare thetone scale modified low resolution digital image for direct viewing. Thefirst transform is a 3×3 color matrix transformation which is applied tothe color pixels of the tone scale modified low resolution digitalimage. The color matrix transformation accounts for the differencebetween the spectral sensitivities of the color photosites of the imagesensing device 10 and the spectral characteristics of the display device64. The method described above employed by the present invention issimilar to the method taught and disclosed in U.S. Pat. No. 5,189,511issued Feb. 23, 1993 to Parulski et al. entitled Method and Apparatusfor Improving the Color Rendition of Hardcopy Images from ElectronicCameras. The second transform involves the application of a tone scalefunction which maps the extended dynamic range pixel values of the tonescale modified low resolution digital image to pixel values compatiblewith typical viewing devices. The present invention uses a similarmethod to the one described in U.S. Pat. No. 5,300,381 issued Apr. 5,1994 to Buhr et al. entitled Color Image Reproduction of Scenes withPreferential Tone Mapping. Buhr describes a method of calculating arendering tone scale function with preferential tone mapping in whichthe contrast of the tone scale function is greatest for midtone pixelintensities and has gracefully lower contrast for shadow and highlightpixel intensities. This rendering tone scale function is combined with agamma mapping tone scale function to calculate a system tone scalefunction. The gamma mapping tone scale function compensates for theinherent intensity response of typical viewing devices. The system tonescale function is cascaded with the rendering tone scale function andapplied, in the form of a look-up-table, to the pixels of the tone scalemodified low resolution digital image resulting in a low resolutionrendered digital image.

For digital camera applications, a display device 64 such as an LCDscreen is used to view digital images produced by the digital imageprocessor 200. For other digital imaging applications the user can alsoview digital images on a display device 64. The user views the imagepresented on the LCD display device and can make selections with regardto the brightness, color and tone. The changes to the brightness andcolor are input to the scene balance module 310 as shown in FIG. 11. Thepresent invention uses the modifications selected by the user to makeadditive changes to the balance parameters. Those skilled in the art ofcolor balance technology will recognize that the present invention canbe practiced with a wide variety of different methods for achievingcolor balance.

A tone scale function for a sparsely sampled extended dynamic rangedigital image can also be generated using the steps of: a) providing asparsely sampled extended dynamic range image sensing device having fastphotosites with a predetermined response to light exposure interspersedwith slow photosites with a slower response to the same light exposure;b) using the image sensing device to produce a sparsely sampled highresolution digital image having fast pixel values produced by the fastphotosites and slow pixel values produced by the slow photosites; c)analyzing the fast pixel values for saturation; and d) if saturation ispresent in the fast pixel values, generating the tone scale functionusing only slow pixel values from the sparsely sampled high resolutiondigital image, otherwise, using only fast pixel values.

The invention has been described in detail with particular reference tocertain preferred embodiments thereof, but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the invention.

PARTS LIST

-   -   2 lens    -   6 optical lowpass filter    -   10 image sensing device    -   14 A/D converter    -   22 DREFA processor    -   26 CFA interpolator    -   28 slow photosite    -   30 fast photosite    -   32 red slow photosite    -   34 green slow photosite    -   36 blue slow photosite    -   38 red fast photosite    -   40 green fast photosite    -   42 blue fast photosite    -   44 slow pixel compensator    -   46 slow pixel thresholder    -   48 fast pixel thresholder    -   50 signal extender    -   51 standard lenslet    -   52 location    -   53 a–d color filter array    -   54 light blocking mask portion    -   55 photosensitive area    -   56 large aperture    -   57 small aperture    -   58 neutral density filter layer    -   59 neutral density filter    -   62 user selection device    -   64 display device    -   200 digital image processor    -   220 paxelization module    -   222 pixel type separator    -   224 paxelization engine    -   232 low resolution module    -   240 enhancement processor    -   310 scene balance module    -   320 ₁ luminance-chrominance module    -   320 ₂ luminance-chrominance module    -   330 ton scale function generator    -   340 ₁ tone scale applicator    -   340 ₂ tone scale applicator    -   350 rendering module    -   360 RGB conversion module

1. A method of generating a tone scale function for a sparsely sampledextended dynamic range digital image, comprising the steps of: a)providing a sparsely sampled extended dynamic range image sensing devicehaving fast photosites with a predetermined response to light exposureinterspersed with slow photosites with a slower response to the samelight exposure; b) using the image sensing device to produce a sparselysampled high resolution digital image having fast pixel values producedby the fast photosites and slow pixel values produced by the slowphotosites; and c) generating the tone scale function using only slowpixel values from the sparsely sampled high resolution digital image. 2.The method claimed in claim 1, further comprising the step of: d)applying the tone scale function to the sparsely sampled high resolutiondigital image to produce a tone scale adjusted sparsely sampled highresolution digital image.
 3. The method claimed in claim 2, furthercomprising the step of: e) generating a full resolution digital imagefrom the tone scale adjusted sparsely sampled high resolution digitalimage to produce a tone scale adjusted full resolution digital image. 4.A computer program product for generating a tone scale function for asparsely sampled extended dynamic range digital image, the computerprogram product comprising computer readable storage medium having acomputer program stored thereon for performing the steps of claim
 3. 5.A computer program product for generating a tone scale function for asparsely sampled extended dynamic range digital image, the computerprogram product comprising computer readable storage medium having acomputer program stored thereon for performing the steps of claim
 2. 6.The method claimed in claim 1, further comprising the steps of: d)generating a full resolution digital image from the sparsely sampledhigh resolution digital image; and e) applying the tone scale functionto the full resolution digital image to produce a tone scale adjustedfull resolution digital image.
 7. The method claimed in claim 6, furthercomprising the step of applying a spatial filter to the full resolutiondigital image.
 8. A computer program product for generating a tone scalefunction for a sparsely sampled extended dynamic range digital image,the computer program product comprising computer readable storage mediumhaving a computer program stored thereon for performing the steps ofclaim
 7. 9. A computer program product for generating a tone scalefunction for a sparsely sampled extended dynamic range digital image,the computer program product comprising computer readable storage mediumhaving a computer program stored thereon for performing the steps ofclaim
 6. 10. The method claimed in claim 1, wherein the image sensingdevice is a color image sensing device having differently coloredphotosites for producing color pixel values and wherein the slow pixelvalues from only one color of photosite are used to generate the tonescale function.
 11. The method claimed in claim 10, wherein thedifferent colors are red, green, and blue and wherein only green pixelvalues are used to generate the tone scale function.
 12. A computerprogram product for generating a tone scale function for a sparselysampled extended dynamic range digital image, the computer programproduct comprising computer readable storage medium having a computerprogram stored thereon for performing the steps of claim
 11. 13. Acomputer program product for generating a tone scale function for asparsely sampled extended dynamic range digital image, the computerprogram product comprising computer readable storage medium having acomputer program stored thereon for performing the steps of claim 10.14. The method claimed in claim 1, wherein the image sensing device is acolor image sensing device having differently colored photosites forproducing color pixel values and further comprising the step of: d)constructing slow luminance pixel values from the different colored slowpixel values; and e) using the slow luminance pixel values to generatethe tone scale function.
 15. A computer program product for generating atone scale function for a sparsely sampled extended dynamic rangedigital image, the computer program product comprising computer readablestorage medium having a computer program stored thereon for performingthe steps of claim
 14. 16. The method claimed in claim 1, furthercomprising the steps of: d) forming a paxelized digital image from theslow pixel values; and e) using the paxelized digital image to generatethe tone scale function.
 17. A computer program product for generating atone scale function for a sparsely sampled extended dynamic rangedigital image, the computer program product comprising computer readablestorage medium having a computer program stored thereon for performingthe steps of claim
 16. 18. The method claimed in claim 1, furthercomprising the steps of: d) generating a low resolution digital imagefrom the sparsely sampled high resolution digital image; e) applying thetone scale function to the low resolution digital image to form a tonescale modified low resolution digital image; f) displaying the tonescale modified low resolution digital image; g) specifying a desiredtone scale adjustment of modified low resolution digital image; and h)employing the specified tone scale adjustment to generate an adjustedtone scale function.
 19. A computer program product for generating atone scale function for a sparsely sampled extended dynamic rangedigital image, the computer program product comprising computer readablestorage medium having a computer program stored thereon for performingthe steps of claim
 16. 20. The method claimed in claim 1, wherein themethod is performed in a digital camera.
 21. A computer program productfor generating a tone scale function for a sparsely sampled extendeddynamic range digital image, the computer program product comprisingcomputer readable storage medium having a computer program storedthereon for performing the steps of claim
 16. 22. The method claimed inclaim 1, wherein the image sensing device is a color image sensingdevice having differently colored photosites for producing color pixelvalues and wherein a tone scale function is generated for each color.23. The method claimed in claim 22, further comprising the steps of: d)generating a full resolution digital image from the sparsely sampledhigh resolution digital image; and e) applying the tone scale functionsto the full resolution digital image to produce a tone scale adjustedfull resolution digital image.
 24. A computer program product forgenerating a tone scale function for a sparsely sampled extended dynamicrange digital image, the computer program product comprising computerreadable storage medium having a computer program stored thereon forperforming the steps of claim
 23. 25. A computer program product forgenerating a tone scale function for a sparsely sampled extended dynamicrange digital image, the computer program product comprising computerreadable storage medium having a computer program stored thereon forperforming the steps of claim
 22. 26. A computer program product forgenerating a tone scale function for a sparsely sampled extended dynamicrange digital image, the computer program product comprising computerreadable storage medium having a computer program stored thereon forperforming the steps of claim
 1. 27. A method of generating a tone scalefunction for a sparsely sampled extended dynamic range digital image,comprising the steps of: a) providing a sparsely sampled extendeddynamic range image sensing device having fast photosites with apredetermined response to light exposure interspersed with slowphotosites with a slower response to the same light exposure; b) usingthe image sensing device to produce a sparsely sampled high resolutiondigital image having fast pixel values produced by the fast photositesand slow pixel values produced by the slow photosites; c) analyzing thefast pixel values for saturation; and d) if saturation is present in thefast pixel values, generating the tone scale function using only slowpixel values from the sparsely sampled high resolution digital image.28. The method claimed in claim 27, further comprising the step of: e)applying the tone scale function to the sparsely sampled high resolutiondigital image to produce a tone scale adjusted sparsely sampled highresolution digital image.
 29. The method claimed in claim 28, furthercomprising the step of: f) generating a full resolution digital imagefrom the tone scale adjusted sparsely sampled high resolution digitalimage to produce a tone scale adjusted full resolution digital image.30. A computer program product for generating a tone scale function fora sparsely sampled extended dynamic range digital image, the computerprogram product comprising computer readable storage medium having acomputer program stored thereon for performing the steps of claim 29.31. A computer program product for generating a tone scale function fora sparsely sampled extended dynamic range digital image, the computerprogram product comprising computer readable storage medium having acomputer program stored thereon for performing the steps of claim 28.32. The method claimed in claim 27, further comprising the steps of: e)generating a full resolution digital image from the sparsely sampledhigh resolution digital image; and f) applying the tone scale functionto the full resolution digital image to produce a tone scale adjustedfull resolution digital image.
 33. The method claimed in claim 32,further comprising the step of applying a spatial filter to the fullresolution digital image.
 34. A computer program product for generatinga tone scale function for a sparsely sampled extended dynamic rangedigital image, the computer program product comprising computer readablestorage medium having a computer program stored thereon for performingthe steps of claim
 33. 35. A computer program product for generating atone scale function for a sparsely sampled extended dynamic rangedigital image, the computer program product comprising computer readablestorage medium having a computer program stored thereon for performingthe steps of claim
 32. 36. The method claimed in claim 27, wherein theimage sensing device is a color image sensing device having differentlycolored photosites for producing color pixel values and wherein thepixel values from only one color of photosite are used to generate thetone scale function.
 37. The method claimed in claim 36, wherein thedifferent colors are red, green, and blue and wherein only green pixelvalues are used to generate the tone scale function.
 38. A computerprogram product for generating a tone scale function for a sparselysampled extended dynamic range digital image, the computer programproduct comprising computer readable storage medium having a computerprogram stored thereon for performing the steps of claim
 37. 39. Acomputer program product for generating a tone scale function for asparsely sampled extended dynamic range digital image, the computerprogram product comprising computer readable storage medium having acomputer program stored thereon for performing the steps of claim 36.40. The method claimed in claim 27, wherein the image sensing device isa color image sensing device having differently colored photosites forproducing color pixel values and further comprising the step of: e)constructing luminance pixel values from the different colored pixelvalues, and f) using the luminance pixel values to generate the tonescale function.
 41. A computer program product for generating a tonescale function for a sparsely sampled extended dynamic range digitalimage, the computer program product comprising computer readable storagemedium having a computer program stored thereon for performing the stepsof claim
 40. 42. The method claimed in claim 27, further comprising thesteps of: e) forming a paxelized digital image from the pixel values;and f) using the paxelized digital image to generate the tone scalefunction.
 43. A computer program product for generating a tone scalefunction for a sparsely sampled extended dynamic range digital image,the computer program product comprising computer readable storage mediumhaving a computer program stored thereon for performing the steps ofclaim
 42. 44. The method claimed in claim 27, further comprising thesteps of: e) generating a low resolution digital image from the sparselysampled high resolution digital image; f) applying the tone scalefunction to the low resolution digital image to form a tone scalemodified low resolution digital image; g) displaying the tone scalemodified low resolution digital image; h) specifying a desired tonescale adjustment of modified low resolution digital image; and i)employing the specified tone scale adjustment to generate an adjustedtone scale function.
 45. A computer program product for generating atone scale function for a sparsely sampled extended dynamic rangedigital image, the computer program product comprising computer readablestorage medium having a computer program stored thereon for performingthe steps of claim
 44. 46. The method claimed in claim 27, wherein themethod is performed in a digital camera.
 47. A computer program productfor generating a tone scale function for a sparsely sampled extendeddynamic range digital image, the computer program product comprisingcomputer readable storage medium having a computer program storedthereon for performing the steps of claim
 46. 48. The method claimed inclaim 27, wherein the image sensing device is a color image sensingdevice having differently colored photosites for producing color pixelvalues and wherein a tone scale function is generated for each color.49. The method claimed in claim 48, further comprising the steps of: e)generating a full resolution digital image from the sparsely sampledhigh resolution digital image; and f) applying the tone scale functionsto the full resolution digital image to produce a tone scale adjustedfull resolution digital image.
 50. A computer program product forgenerating a tone scale function for a sparsely sampled extended dynamicrange digital image, the computer program product comprising computerreadable storage medium having a computer program stored thereon forperforming the steps of claim
 49. 51. A computer program product forgenerating a tone scale function for a sparsely sampled extended dynamicrange digital image, the computer program product comprising computerreadable storage medium having a computer program stored thereon forperforming the steps of claim
 48. 52. A computer program product forgenerating a tone scale function for a sparsely sampled extended dynamicrange digital image, the computer program product comprising computerreadable storage medium having a computer program stored thereon forperforming the steps of claim
 27. 53. A system for generating a tonescale function for a sparsely sampled extended dynamic range digitalimage, comprising: a) a sparsely sampled extended dynamic range imagesensing device having fast photosites with a predetermined response tolight exposure interspersed with slow photosites with a slower responseto the same light exposure for producing a sparsely sampled highresolution digital image having fast pixel values produced by the fastphotosites and slow pixel values produced by the slow photosites; and b)means for generating the tone scale function using only slow pixelvalues from the sparsely sampled high resolution digital image.
 54. Thesystem claimed in claim 53, further comprising: c) means for applyingthe tone scale function to the sparsely sampled high resolution digitalimage to produce a tone scale adjusted sparsely sampled high resolutiondigital image.
 55. The system claimed in claim 53, further comprising:d) means for generating a full resolution digital image from the tonescale adjusted sparsely sampled high resolution digital image to producea tone scale adjusted full resolution digital image.
 56. The systemclaimed in claim 53, further comprising: c) means for generating a fullresolution digital image from the sparsely sampled high resolutiondigital image; and d) means for applying the tone scale function to thefull resolution digital image to produce a tone scale adjusted fullresolution digital image.
 57. The system claimed in claim 56, furthercomprising means for applying a spatial filter to the full resolutiondigital image.
 58. The system claimed in claim 53, wherein the imagesensing device is a color image sensing device having differentlycolored photosites for producing color pixel values and wherein the slowpixel values from only one color of photosite are used to generate thetone scale function.
 59. The system claimed in claim 58, wherein thedifferent colors are red, green, and blue and wherein only green pixelvalues are used to generate the tone scale function.
 60. The systemclaimed in claim 53, wherein the image sensing device is a color imagesensing device having differently colored photosites for producing colorpixel values and further comprising: c) means for constructing slowluminance pixel values from the different colored slow pixel values; andd) means for using the slow luminance pixel values to generate the tonescale function.
 61. The system claimed in claim 53, further comprising:c) means for forming a paxelized digital image from the slow pixelvalues; and d) means for using the paxelized digital image to generatethe tone scale function.
 62. The system claimed in claim 53, furthercomprising: c) means for generating a low resolution digital image fromthe sparsely sampled high resolution digital image; d) means forapplying the tone scale function to the low resolution digital image toform a tone scale modified low resolution digital image; e) means fordisplaying the tone scale modified low resolution digital image; f)means for specifying a desired tone scale adjustment of modified lowresolution digital image; and g) means for employing the specified tonescale adjustment to generate an adjusted tone scale function.
 63. Thesystem claimed in claim 53, wherein the system is in a digital camera.64. The system claimed in claim 53, wherein the image sensing device isa color image sensing device having differently colored photosites forproducing color pixel values and wherein a tone scale function isgenerated for each color.
 65. The system claimed in claim 64, furthercomprising: c) means for generating a full resolution digital image fromthe sparsely sampled high resolution digital image; and d) means forapplying the tone scale functions to the full resolution digital imageto produce a tone scale adjusted full resolution digital image.
 66. Asystem for generating a tone scale function for a sparsely sampledextended dynamic range digital image, comprising: a) a sparsely sampledextended dynamic range image sensing device having fast photosites witha predetermined response to light exposure interspersed with slowphotosites with a slower response to the same light exposure forproducing a sparsely sampled high resolution digital image having fastpixel values produced by the fast photosites and slow pixel valuesproduced by the slow photosites; b) means for analyzing the fast pixelvalues for saturation; and c) means for generating the tone scalefunction using only slow pixel values from the sparsely sampled highresolution digital image if saturation is present in the fast pixelvalues.
 67. The system claimed in claim 66, further comprising: d) meansfor applying the tone scale function to the sparsely sampled highresolution digital image to produce a tone scale adjusted sparselysampled high resolution digital image.
 68. The system claimed in claim67, further comprising: e) means for generating a full resolutiondigital image from the tone scale adjusted sparsely sampled highresolution digital image to produce a tone scale adjusted fullresolution digital image.
 69. The system claimed in claim 66, furthercomprising: d) means for generating a full resolution digital image fromthe sparsely sampled high resolution digital image; and e) means forapplying the tone scale function to the full resolution digital image toproduce a tone scale adjusted full resolution digital image.
 70. Thesystem claimed in claim 69, further comprising means for applying aspatial filter to the full resolution digital image.
 71. The systemclaimed in claim 66, wherein the image sensing device is a color imagesensing device having differently colored photosites for producing colorpixel values and wherein the pixel values from only one color ofphotosite are used to generate the tone scale function.
 72. The systemclaimed in claim 71, wherein the different colors are red, green, andblue and wherein only green pixel values are used to generate the tonescale function.
 73. The system claimed in claim 66, wherein the imagesensing device is a color image sensing device having differentlycolored photosites for producing color pixel values and furthercomprising: d) means for constructing luminance pixel values from thedifferent colored pixel values; and e) means for using the luminancepixel values to generate the tone scale function.
 74. The system claimedin claim 66, further comprising: d) means for forming a paxelizeddigital image from the pixel values; and e) means for using thepaxelized digital image to generate the tone scale function.
 75. Thesystem claimed in claim 66, further comprising d) means for generating alow resolution digital image from the sparsely sampled high resolutiondigital image; e) means for applying the tone scale function to the lowresolution digital image to form a tone scale modified low resolutiondigital image; f) means for displaying the tone scale modified lowresolution digital image; g) means for specifying a desired tone scaleadjustment of modified low resolution digital image; and h) means foremploying the specified tone scale adjustment to generate an adjustedtone scale function.
 76. The system claimed in claim 66, wherein thesystem is in a digital camera.
 77. The system claimed in claim 66,wherein the image sensing device is a color image sensing device havingdifferently colored photosites for producing color pixel values andwherein a tone scale function is generated for each color.
 78. Thesystem claimed in claim 77, further comprising: d) means for generatinga full resolution digital image from the sparsely sampled highresolution digital image; and e) means for applying the tone scalefunctions to the full resolution digital image to produce a tone scaleadjusted full resolution digital image.